Overview of the Informational Retrieval Task at NTCIR-4 WEB

This paper gives an overview of the Informational Retrieval Task 2 that was conducted from 2003 to 2004 as a subtask of the WEB Task at the Fourth NTCIR Workshop (‘NTCIR-4 WEB’). In the Informational Retrieval Task, we attempted to assess the retrieval effectiveness of Web search engine systems from a viewpoint of topical relevance, and to build a re-usable test collection suitable for evaluating Web search engine systems from such a viewpoint. We used a 100gigabyte document dataset that was mainly gathered from the ‘.jp’ domain. Relevance judgments were performed on the retrieved documents written in Japanese or English, partially considering the relationshiop between the pages referenced by hyper-links. We also investigated the evaluation methods considering nonredundancy of contents and diversity of queries.